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ABSTRACT 

Criticism is the art of disclosing the qualities of 
events or objects the connoisseur perceives. Criticism is not 
negative appraisal, nor a list of unfavorable judgements; its essence 
is illumination. The key characteristics of criticism used as an 
evaluation method are: (1) inherent subjectivity; (2) reliance on the 
perception of the critic (its quality is a function of that person's 
ability); (3) descriptive, interpretative, and evaluative qualities; 
and (4) intent to educate others so that their appreciation and 
understanding of the critiqued work will be enhanced. Criticism 
techniques should be used when: (1) the purpose of the study is to 
convey to others the nature and qualities of a program or a set of 
materials; (2) an expert opinion can illuminate and educate others 
about the program; (3) the intent of the evaluation is to discover 
what the program is, how it is what it is, rather than what it does; 
and (4) a subjective and qualitative appraisal of a program is 
satisfactory. ( JAZ) v 9 



* Reproductions supplied by EDRS are the best that can be made * 

from the original document. * 
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The function of criticise should be to show 
how a work is what it is, even that ifcis what 
it is, rather than to show what it means. 

Susan Son tag 

I'd rather learn from one bird how to sing, 
than teach ten thousand stars how not to dance. 

e. e. cunnings 



INTRODUCTION 



As a brief introduction to using criticism methods in 
evaluation, following are brief lists of (a) the key characteris- 
tics of criticism, and (b) hints about when to use the techniques 
of criticism. 

Key characteristics of criticism include the following, criticism 
e is inherently subjective; 

• relies on the perception of the critic, and its 
quality is a function of that person's ability; 

e is descriptive, interpretative, and evaluative; 

e aims to educate others so that their appreciation and 
understanding of the critiqued work will be enhanced. 

use the techniques of criticism uhen 



e the purpose of the study is to convey to others the 
nature and qualities of a program or a set of 
materials; 

e an expert or connoisseur opinion can illuminate and 
educate others about the program; 

• the intent of the evaluation is to discover what the 
program is, how it is what it is, rather than what it 
does; 



a subjective and qualitative appraisal of a program is 
satisfactory. 



THE CRITIC AS A GUIDE 



In The Ascent of Man, Jacob Bronowski, scientist, 
mathematician, and philosopher, guides all those who will follow 
over an unfamiliar landscape. Speaking to the common man, he 
gently works his way through the history of science and tech- 
nology, pausing to examine in detail pivotal times, events, 
people, and ideas. In describing the invention of the 
Pythagorean theorem, or Einstein's pondering a about relativity, 
it would have been easy to lose the interest of all but the most 
ardent science enthusiasts. Yet, Bronowski managed to produce 
one of the most popular television science series of all times. 



Bronowski manages to illuminate the most difficult ideas in 
science in an intriguing, even poetic, fashion. Articulate, 
facile, and clearly at home as he talks about the great 
scientists of the past, he helps his listeners to see the 
brilliance of their insights, to appreciate the subtleness of 
their theories, and to recognise the deeper implications of 
their work. 



More importantly, Bronowski loves science. He is proud to 
be a scientist. It is obvious that he relishes the intellectual 
struggle that is at the heart of doing good science. His 
intensity and earnestness move his listeners as much as his 
intelligence. 



Bronowski is a great critic. 



TUB NATUHB OP CRITICISM 



The end of criticism is the re-education of 
the perception of the work of art. The task 
of the critic is to lift the veils that keep 
the eyes from seeing. 

Dewey 



Connoisseurship 

Blliot Bisner, in his book The Educational Imagination 
(1979) , describes connoisseurship in this ways 

To be a connoisseur is to know how to look, to see, and 
to appreciate. Connoisseurship, generally defined, is 
the art of appreciation. It is essential to criticism 
because, without the ability to perceive what is subtle 



and important, criticism is likely to be superficial or 
even empty. The major distinction between connoisseur- 
ship and criticism is this: connoisseur ship is the art 
of appreciation, criticism is the art of disclosure. 
Connoisseur8hip is a private act; it consists of 
recognising and appreciating the qualities of a 
particular, but it does not require either a public 
judgment or a public description of those qualities . • • 



Connoisseurship comes from experience and attention. To be a 
connoisseur of wine, one must have had considerable experience in 
tasting wines and in being attentive at the time. Experience 
alone obviously does not make one a connoisseur (or else there 
would be a significant glut of wine connoisseurs) . One has to 
learn what qualities exist, how they are sensed, and how they 
relate. One must have sensory benchmarks in memory— to remember 
how other great wines compare in these qualities. 



One can be a connoisseur of anything— wine, science, music, 
film, old cars. . • wine connoissuers have a trained and refined 
sense of taste and smell; music connoisseurs have trained ears; 
restaurant connoisseurs have both trained palates and are 
sensitive to ambiance and mood. Educational connoisseurs are 
sensitive to teaching and learning behaviors. Just as with music 
or art, educational connoisseurship results from a combination of 
experience and attention. Educational connoisseurs must have 
seen a wide range of educational settings, have observed students 
and teachers interacting, and have studied curricula and 
educational materials. Prom such experience, if attention is 
present, sensitivity can be gained. 



Connoisseurs love what they know. Their sensitivity comes 
from their appreciation. Their learning is fired by their 
passion for their subject. Music connoisseurs like to listen to 
music; they think music is important to their lives; they believe 
in music. Similarly, educational connoisseurs must have a 
natural interest in education, in learning, in ideas, and in the 
development of people. 



Criticism 

Criticism is the art of disclosing the qualities of 
events or objects the connoisseur perceives. Criticism 
is the public side of connoisseurship. One can be a 
connoisseur without the skills of criticism, but one 
cannot be a critic without the skills of connoisseurship. 

Eisner (1979) 



Bronowski is both a connoisseur and a critic. Ha is both 
knowledgeable and articulate. He does what the critics of art, 
wine, and music do—he illuminates the whole field for us. He 
allows us to see fcr ourselves. 



Criticism points out structures to us. it notes repetitive 
themes and maktts patterns visible, it makes what might be 
originally an indistinguishable blur into a picture with 
distinguishable features. Our understanding of how a work is 
unique, how it gains its integrity and brilliance, is enhanced. 
Good criticism allows us to see deeper, to notice more, and to 
have quesions (more difficult questions) of our own. it brings 
us closer to the artist, and makes the artistic enterprise more 
real, familiar, and feasible. 



Critics help us to see what is significant in a work. They 
share with us what they perceive. And their perceptions are a 
function of their concepts. That is, what they perceive depends 
upon how they look, which in turn depends upon their training and 
knowledge. Critics use constructs and theories to guide their 
vision. (The word theory, in fact, coses from the Greek word for 
theater—thus, a theory is a stage, a reference frame for viewing 
the world.) For example, the theories of piaget may be used to 
help us understand a classroom where children are learning 
elementary science ideas, or the theories of social psychology 
may help the critic in showing us the important aspects of the 
class management techniques being used. Eisner explains in more 
detail how the educational critic chooses what to relate: 

What (educational) critics do or should try to do is not 
to translate .what cannot be translated, but rather to 
create a rendering of a situation, event, or object that 
will provide pointers to those aspects of the situation, 
event, or object that are in some way significant, now 
what counts as significant will depend upon the theories, 
models, and values (of the critic) . But it will also 
depend on the purpose of the critic, por example, an 
educational critic focusing on the learning patterns of a 
particular student will attend to qualities and 
circumstances different from those he or she would attend 
if interested in the critical rendering of the character 
of classroom discourse, the qualities of the classroom's 
visual environment, or the meanings imbedded in the 
treatment of time in the classroom. The point here is 
straightforward: what is rendered by someone working as 
an educational critic will depend on his or her purposes 
as well as the kinds of maps, models, and theories being 
used. 

To complete this general discussion of the nature of criticism, 
it may be useful to point out what criticism is not. 



Criticism is not negative appraisal, it is not a list of 
unfavorable judgments. Bather, the essence of criticism (at 
least, as we are describing it in this guide) is illumination. 
Criticism involves description, interpretation, and evaluation. 
All criticism involves description, which hopefully aids ua in 
seeing the work more clearly. Criticism also involves 
interpretation—which is usually based on some sort of 
theoretical framework. Finally, criticism may involve some 
degree of evaluative judgment, depending on the purpose of the 
evaluation. A critic who is writing for Consumer Reports may 
want to make all kinds of comparative judgments. (The critic 
may, in tact, be "critical".) On the other hand, there are many 
times that description and interpretation are enough. For 
example, the critic who is writing to enlighten visitors a* an 
art gallery may want to make few comparisons or judgments. 



Criticism is also not classification. Recognizing the age of 
a wine, the year of a painting, or the Piagetian stage of a child 
is not the aim of the critic. Good criticism relies on fresh 
perceptions which may be actually blocked by classification. The 
difference between classification and perception has become very 
clear to me in the pursuit of my hobby of bird watching. Mostly, 
I do not like organized bird-watching trips. The guides usually 
are skilled and knowledgeable, but they often offer little more 
than assistance with observation and classification. The trip 
becomes an endless series of sightings and classification. A 
friend of mine. Bill Clow, is a unique guide by contrast. Billy 
is a connoisseur of birds and a great critic of bird-watching. 
First, he is in love with his occupation— all of his energy goes 
into understanding and appreciating birds. And he shares that 
appreciation. Bis passion is obvious and infectious. He points 
out where birds live, and why; he mocks their calls; he explains 
their features! he talks with owls at night, with Billy, you do 
not just learn a taxonomy— you enter a whole new world. 



An Example of Educational Criticism 

One can easily find reviews of be oka and movies, but it is 
more difficult to find examples of educational criticism. The 
following passages, excerpted from an educational criticism by 
Elizabeth Vallance, are included in The Educational Imagination 
(Eisner, 1979). This particular section deals with the quality 
and types of intellectual discussions happening in Mr. Williams 1 
classroom. 

Mr. Williams: I wonder why the ocean never fills up if all 
these rivers keep flowing into it. 

J*** The ocean goes on and on— it never fills up. 

It's very big. 

Sue: People take water out of it to drink, in 

reservoirs. 
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Mr. William* (surprised): To drink it!?.' 

Jack: it evaporates and lotsa people use it to drink. 

Mr. Williams: water from the ocean? 

Jack (flustered, correcting himself): Oh, no. it's salty. But 
it evaporates and then it rains. . . 

Tn this discussion. Nr. Williams does not fully resolve the 
question by answering it precisely, making it possible for the 
chidren to continue to wonder about it. Additionally, he does 
not restate Jack's final point, perhaps forcing chidren to listen 
to one another if they are to enter tLe discussion. . . Leaving 
some questions unresolved may demonstrate that every time one 
wonders about something, one should not expect to find an 
answer. . . 

Later Mr. Williaaa askst what do you think the author had in 
mind when he wrote this scene? what was the difference between 
the world Mole lived in and the one water Rat lived in? Do 
people live in different worlds? 

Reading a phrase, he points out that the author of wind in 
the willows makes the river have life, describing it as an animal 
playing with leaves, chuckling. 



Mr. Williams: Did you ever sit by a river? 

Jiai It gurgles, (others murmur assent) 

Mr. Williams: You can have a conversation with it. 



Moving the questions among students skillfully, so all have a 
chance to answer in front off the class, Mr. Williams is not 
willing to accept just any answer. He may tell a student to 
reread a section, to find evidence, to clarify the meaning of 
what was said, to amplify an interpretation or to rethink a 
position. And he may sum up previous explanations before askit a 
for more plausative alternatives, while this could conceivably 
deflate some children and prevent them from being able to respond 
to another question, in Mr. Williams' class, children do not see. 
to be particularly bruised by making mistakes in front of their 
peers and by being told their answers are wrong. . . it is a 
supportive group, they do not ridicule one another's mistakes, 
and Mr. Williams generally lata students know their answers are 
wrong in a matter-of-fact tone, not accusingly or sending 
messages that their answers are stupid. Another child reads a 
story and is trying to interpret a passage. 
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Nr. Williams: why did Oswald say that his legs are younger than 
Grandpa 1 a? 

Holly: Because he is younger than Grandpa? 

Nr. William* i I think that there is nore to it than that. 

He communicates clearly, but without recrimination, that the 
answer is incomplete, there are, after all, right and wrong 
answers, complete and incomplete ones, and central and peripheral 
contributions to a discussion. It seems more honest and 
responsible to let children know that their answers are wrong in 
a supportive setting than to go along willy-nilly and accept any 
and all answers. • • 



This example shows the major features of criticism. It is 
descriptive and interpetative. It is even, at times, evaluative. 
The critic* a love of good teaching is apparent. The brief 
classroom discussions that are included illustrate themes that 
run throughout the class. Theories about psychological 
well-being and behavior lie behind the interpretations that are 
suggested. Meaning is extracted (or added) to very brief 
glimpses. This criticism is, as is all criticism, subjective. 
Even the bias of tto critic for open-ended, probing, and less 
structured situations is apparent. 



THE TKHmQGBS OP TBB CRITIC 



In this sec* on of the guide, four specific working 
techniques of the critic are described. The techniques range 
from having direct and obvious correlates in program evaluation 
to only suggesting new and creative possibilities. 



Accurate, Sharp, Appreciative Descriptio ns 

In the film Amadeus, Saliere, an arch-rival of Mozart's, 
reflects jealously on the brilliance of Hosart's music. As he 
describes the unique structures and movements, the symphyony 
becomes audible in the background. Then Salieri's words are 
blended with the music, and he lovingly points out the entrance 
of each instrument, the novelty of its role, and its place in the 
whole of the symphony. He sighs, feeling each nuance of the 
music's mood, and he talks of Nosart, his life, and how the music 
is only a very honest, indeed ruthless, statement by Nosart of 
the way it is. 
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Salieri 's description is itaolf a piec* of art and a 
magnificent piece of criticise. She listener learns about the 
music, and about Mozart, but is also brought directly into 
contact with the transcending qualities of all music. The 
listener is uplifted by the experience. 



This is a good exaqple of an accurate, sharp, loving 
description. The example of educational criticism written by 
Elisabeth Vallance is another example. This technique 
intertwines the comments of the critic with the flow of the work 
itself, so that the listener can learn about the work as it is 
happening. Voice-overs on film, documentary narrators, and even 
sports commentators, all use this technique. 



In the final example given below, Randall Jarrell (1965) uses 
this technique to comment on Robert Frost's dramatic poem, "Home 
Burial;" 

He saw her from the bottom of the stairs 
Before she saw him 

(implies that, knowing herself seen, she would have 

acted differently) 

She was staring down 

looking back over her shoulder at some fear 

(at soae fear. • • not a specific feared object. • • 
Normally, we do not look back over our shoulder at 
what we leave, unless we feel for it something more 
than fear.) 

She took a doubtful step 

and then undid it 

To raise herself and look again. 

(A vertical ballet of indecision toward and away from 
a fearful, but mesmerically, attractive object, 
something hard to decide to leave, and easy to decide 
to return to. • • The surprising use of the word 
"undid" gives her withdrawal of the tentative step a 
surprising reality.) 

And Jarrell shows how the style of the poem supports the mood 
of the works 

If you had any feelings, you that dug 

with your own hand 

(but after the three stabbing, indicting stresses of 
"your-own-hand," she breaks off the sentence. • • but 
the fact of things continues) 

I saw you from that very window there 

(the same stabbing stresses, the sane emphasis of a 
specific damning actuality. . .) 
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You could sit there with the stains on your shoes 
of the fresh earth from your own baby's grave 
And talk about your everyday concerns 

(The hissing or splitting a's together with the 
damning stresses and the judge* a summing up give an 
awful finality to this condemnation. Later, she says 
that he lives by proverbs and he'll die by then. Now 
she turns on him and in breathless outrage.) 

Think of it, talk like that at such a time! 

(the repeated sounds of t, t, th, t, t are thoroughly 
expressive) 

Jarrell is not following any set rules of criticism. He is 
simply getting into the work, staying close to what he hears and 
feels and sees. He is writing out a blow-by-blow description of 
the unseen, but richly emotional action, that surrounds the lean 
verses. 



In an educational setting, this kind of description can be 
used to critique classroom interactions, teaching styles, and 
educational materials. Such a description will be only as 
insight !ul as the critic is sensitive and perceptive about 
educational issues. 

Thematic Matrix Ana lv a la 

One of the critic's tasks is to find and point out the 
dominant themes or patterns that run through a aork. Thematic 
matrix analysis is a procedure that critics use to dissect a 
work, to record the themes that are present, and to study bow 
they interact. 

A matrix is a two-dimensional array that shows interactions 
between two sets of data (the column and row variables) . it is 
used as an exploratory tool, as well as a record keeping and 
display device. Matrices can be used to write down the way that 
themes or issues are used in a work. They are particularly good 
issues rtiR9 dichotOBle8 ' tor comparing and contrasting polar 

Example < 

The Evaluation of the Follow Through Program drew 
critiques from two •valuators-- House and Stebbins. 
Their reports were independent and difficult to 
reconcile. The matrix on the following page pulls out 
the prominent themes or points where comparison is 
important. 



11 



10 



Theme 

1. The measures lack 
sensitivity to the unique 
outcome of the different 
models 

2. The proper unit of 
analysis is the classroom 
rather than the child 

3. Models emphasizing 
basic skills succeeded 
better than other models 
in helping children gain 
these skills 



4. Program implementation 
was not assessed 




Stebbins 



Agrees 



Agrees 



Disagrees 



Agrees 



Disagrees in 
text, not in 
summary 



Agrees 



Agrees 
(implicitly) 



Matrices can be used to decide whether a theme is really a 
theme. That is, when an idea or issue begins to appear before 
the evaluator repeatedly, it may be something the evaluator 
wishes to keep track of and document more fully. The following 
matrix can be used by an evaluator to keep track of recurring 
themes encountered in a qualitative evaluation. 



Theme 

1. Participants are 
good peer teachers 

2. The goal of the 
workshop is not to 
improve teaching 
directly 



Interview Observation 
55.2-56.0 



Document 



P1.3 
K3.7 

63.5 



42.2 
65.0 



proposal 
23.3 



The themes, ideas, or issues of the program being studied are 
listed in the left-hand column. As participants mention these 
themes in interviews, they are noted on the chart. (All inter- 
views are typed into transcripts and the pages labeled — thus F1.3 
is three-tenths of the way down the first page of interview F.) 
When the same theme shows up in the field notes (a page numbered 
notebook) , the location is noted in the appropriate cell of the 
matrix. Finally, when the issue appears in a document, question- 
naire, or written materials, its occurrence is also noted. 



With the advent of powerful microcomputer data based 
management systems, it should be possible to store and cross- 
reference a large amount of rich qualitative data. 
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•tea— of Co n s ci ousnaM 



Zn Virginia Wolfe's To tha LlahthouM . tha narrator of tha 
story ten alaoat ooaplateiy vaniahad, and tbo raadar sots lnataad 
of objaotlva raality, only its roflootion in tha oyoo ano alnda 
of tha oharaotera* To giva prooadanoa to tha internal raality of 
oonaolousnoss ovar tha asternal raality of facta and actions is a 
taohaiqua that authors and orltioa hava usad for good affaot. 
This tachnlqua of viawlng a aoana--of not aaaing it dlraotly, but 
only aaaing its raf laotions in tha oonaoiousnaaa of thoaa who ara 
thara ■ «haa tha following oharaoteriatiost 

1. Objaotlva foots and notions ara aaan only aa alaMnta 
in tha Binds and eyaa of tha oharaotera. Such 
aubjactlva vlawa of raality aay ho inconsistent, and 
thay angaga us in wondaring about tha natura of an 
'objaotlva* raality which wa cannot aaa for ouraalvaa. 

2. Tha raality of apaoa and tiM aay ba violated in 
ordar to praaant a olaar viaw of tbo innar raality. 
Thus, aoasnts aay axpand and flashbacks My juap 
larga gapa in spaoa and tiM. 



As an axaapla, ooMidar tha da script ion of a houaa givan 
in To tha LiahthouMt 

•ha looted up • • • and saw tha room, saw tha chalra, 
thought tbaa fMrfully shabby. Thair antraila, aa Andrew 
Mid tha othar day, wara all ovar tha floor i but than 
what wm tha point, aha askad harMlf , of buying good 
chairs to 1st ttaa spoil up hara all through tha winter 
whan tha houM, with only ooa woaan to aaa it, positively 
drippad with wot? 

Wa aaa tha houM not "aa it ia" but only aa it ia in urn. 
Haaesy'e ayaa. wa notloaa only what aha notlcaa. wa laarn not 
only about tha houM, but about lira. Raaeey. 



Par a valuator a, tha taohniqua of focusing on tha internal 
raalitlM of tha oharaotera in tha Mtting can ba vary 
illuminating* TO deaoribe a prog ran, ona can usa tha daaorlptlcn 
of aa •objaotiva narrator" or ona can praaant tha program aa it 
ia raf looted ia tha ayaa of its atudanta, teachara, administra- 
tors, sad f under s. Tha literary techniques axiat for graphically 
illustrating tha dlffarant prograa raalltiaa for aach. 



DissoatinB itias in Word and Iaaga 

Two film tochnlgoea ara UMful to tha a valuator in asking 
aubtla oritioal ccaaantary. 
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Lateral tracking is a technique where the camera aoans a 
sequence o£ somas that ara not naoassarily continuous in space 
or t is*, but are connected, perhaps subtly, by iom these. Thus, 
alternating scenes of poverty and opulence can sake a powerful 
social eosaentary without any direct narrative. Siailarly, a 
sequence of scenes of an eduoational program can. by itself, b* 
an effective tool of criticise. * 

Another type of effective discontinuity is one between word 
and isage. Nhen program goal etateaente are diepluyed 
simultaneously with program realities, an illuminating discord or 
her sony can be shown. 



Delle-Piana, G. M. (1981) . Literary and film criticism, in 
Smith, N . (Bd.), Metaphors for evaluation . Beverly Hilla. 
Cat Sage Publications. 

Bisner, b. W. (1979). The educati onal <— o^— <~ Hew York: 
MacBlllan. 

Jarrell, R. (1965). The third book of criticim . New York i 
Farcer, Straus and Giroux. 
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